DOCUMENT RESUME 



ED 068 641 



VT 016 875 



AUTHOR 

TITLE 

INSTITUTION 

SPONS AGENCY 
NOTE 



Las ell, Warren L. ; And Others 

A Model to Evaluate Inservice Personnel Development 
in Vocational-Technical Education. 

Ohio State Univ. , Columbus. Center for Vocational and 
Technical Education. 

Office of Education (DHEW) , Washington, D.C. 

82p. 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



MF-$0.65 HC-$3.29 

Behavioral Objectives; Coordinators; Guidelines; 
♦Inservice Teacher Education; Models; *Personnel 
Evaluation; Program Administration; ♦Program 
Evaluation; ♦Staff Improvement; ♦State Programs; 
State Surveys; Vocational Education Teachers 
♦Education Professions Development Act; EPDA 



ABSTRACT 

The evaluation model described in this document was 
developed to provide States with one means of evaluating personnel 
development programs and projects. It was designed to supplement 
other State evaluation efforts by presenting guidelines and materials 
for use by State coordinators and evaluation teams in evaluating 
State inservice personnel development in vocational-technical 
education funded by the Federal and State governments. The purpose of 
the State evaluation team is to analyze data, make recommendations, 
and prepare a final evaluation report for the State director of 
vocational education. The model and procedures described in the 
materials may be modified by individual States to better suit their 
programs. The evaluation model package consists of nine documents; 
overview; guidelines for developing behavioral objectives; three 
instruction manuals for the State coordinator and evaluation team; 
and four survey forms. The survey forms are for the evaluation team's 
use in gathering data about the State personnel development program 
and individual projects. The manuals provide instructions for use of 
the survey forms, for preparation of information from the surveys, 
and for conducting meetings and interpretating data. (MF) 
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Congress recognized the need for better trained educators when it 
passed the Education Professions Development Act of 1965. Three years 
later, it amended the law in the 1968 Vocational Education Amendments. 
Sections 552 and 553 of Part F of the amendments authorized programs 
to: (l) grant doctorate degrees in vocational education and (2) provide 

inservice training for vocational educators already on the Job. Programs 
funded under Section 552 involve full-time study leading to the doctorate; 
those under Section 553 typically involve part-time study, such as inservice 
education. 

The Model to Evaluate Inservice Personnel Development in Vocational- 
Technical Education is concerned with inservice personnel development 
funded by federal and state governments. It is designed for activities 
funded under Section 553 of the 1968 amendments and those similar to 
Section 553 projects but not funded from that source. It is not intended 
for use with either pre-service personnel development programs or projects 
or with programs involving full-time graduate study. 

As states have begun personnel development activities, the need to 
evaluate programs and projects has become apparent. In addition, the 
current accountability movement in American education has increased 
pressures upon states to evaluate all programs and projects. 

The evaluation model described in this document was developed to 
provide states with one means to evaluate personnel development programs 
and projects. It is designed to supplement other state evaluation efforts, 
including the evaluation of individual projects, by providing states with 
more information about individual projects and their total personnel 
development program. The model is described and definitions are given in 
the remainder of this overview. 



A Description of the Evaluation Model 

The model includes a set of six written materials and several pro- 
cedures described in the materials. The materials and several procedures 
are described briefly below. 

Materials of the Model 

1. Overview . The overview is this document. It describes the 
materials and procedures of the evaluation model and defines 
several terms essential to the operation of the model. 
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2. Manual of Instructions for the Distribution and Collection of 
Survey Forms and Selection of the State Evaluation Team (M-lJ. 

The manual provides the state coordinator of personnel development* 
with instructions to use four survey forms from the model to 
gather information about the state program and individual projects 
and instructions to select a state evaluation team (SET). 

3. Survey Forms. The model includes survey forms to gather data 

from: (a) the state coordinator of personnel development, 

(b) directors of personnel development projects, (c) participants 
of personnel development projects, and (d) supervisors of 
personnel development participants. 

4. Manual of Instructions to Prepare Data for the State Evaluation 
Team Meeting (M-2). The document provides the state coordinator 
of personnel development with instructions and a format to prepare 
information from the four surveys for use by members of the state 
evaluation team (SET). 

5. Manual of Instruction for the State Evaluation Team Meeting (M-3). 
The manual provides members of the state evaluation team with 
instructions to conduct the SET meeting and to interpret infor- 
mation prepared for the SET by the state coordinator. It includes 
an appendix with a description of the state coordinator’s respon- 
sibilities for the SET meeting. 

6. Guidelines for Developing Behavioral Objectives . The guidelines 
are designed to provide those responsible for personnel develop- 
ment programs and projects with assistance in writing behavioral 
objectives. 



Procedures of the Model 

Because it is to be used by individual states for their benefit, the 
model may be modified to better suit the needs of a state's personnel 
development program. A state may decide, for example, not to use sore 
of the materials of the model. Or, it may change procedures outlined in 
the materials. In either case, the procedures would then differ from 
those suggested. The suggested procedures, described generally in the 
paragraphs that follow, assume the full use of all materials of the model. 



*The title used in this model to refer to the person primarily 
responsible for administering the state personnel development program 
in vocational-technics!, education. Some states use instead titles 
such as Part F coordinator and state director of personnel development. 
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The model, as described herein, also assumes that the state coordinator 
of personnel development will occupy a key role in the evaluation. If 
the state director of vocational education assigns the responsibility for 
the use of this evaluation model to another person, the procedures which 
follow should be undertaken by the person with that responsibility. 

Detailed instructions needed by the state coordinator to perform his 
duties are given in manuals M-l, M-2, and M-3. The first function is to 
collect data (M-l). The state coordinator should determine the number of 
personnel development projects from which to collect data and then determine 
a sample of participants from those projects. 

When these decisions on sampling have been made, the coordinator should 
distribute the model? s three survey forms to collect data from (l) directors 
of personnel development projects, (2) participants of projects, and 
(3) their sr;»ervisors. In addition, he should complete the State Program 
Coordinator Survey. 

The coordinator should use M-l to select five persons, including a 
chairman, to serve on the SET. He should schedule a two- or three-day 
meeting for the SET members and himself. 

Using manual M-2, the state coordinator should prepare information 
collected with the surveys for members of the SET. If there is a sufficient 
number of projects within the state, he may elect to use the computer 
program in Appendix 2 of M-2 to process the data. 

Prior to the SET meeting, the coordinator should send team members 
information about the meeting and their functions at it. He should not 
send program and project data to the team before the meeting. Team 
members should receive that data during the meeting. 

The SET should use information received before the meeting, the 
project and program data folders (explained in M-2) provided at the 
meeting by the state coordinator, and whatever additional data its 
members need and can get to prepare an evaluation report (M-3). The 
chairman of the SET, appointed by the state coordinator, should be 
responsible for conducting the meeting of the team, making work assignments 
for team members, and insuring that the evaluation report is given to the 
state director of vocational education. 

The state director, upon receiving the evaluation report, should 
review it and take whatever action he deems necessary. The evaluation 
process should be recycled regularly or sporadically, at the discretion 
of the state director of vocational education. The procedures of the 
evaluation are shown graphically on the page which follows. 
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Development of the Model 

The model was developed in two stages. In the first stage, a 
conceptual design and a draft of the model were prepared. Numerous 
consultants and reviewers assisted in their development. Some 20 persons 
provided the model developers with written reactions to early drafts of 
the model. These reactions were used to develop subsequent drafts of the 
model. 

The second stage of development was a pilot test of the model in 
two states, Tennessee and California. Results of the test were of great 
help in improving the model. During the test, state coordinators received 
all materials of the model except instructions to process data in M-2. 

The model developers processed data and prepared information in the 
required format for SET meetings. The state coordinators followed all 
other procedures in the model, without help from the developers. State 
coordinators in California and Tennessee each selected three persons 
to serve on the SET. In both states, the SET prepared an evaluation 
report within two days. The developers of the model observed the operation 
to improve the model. 
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Definitions 

Participant — A person who has participated in a project designed to 
increase his or her effectiveness and efficiency in vocational education. 
Participants maybe teachers, coordinators, administrators, or others 
involved in personnel development projects. 

Personnel Development Program — All personnel development projects in 
the state. 

Personnel Develop ment Project — A component of the state personnel 
development program. As used here, project does not designate pre- 
service teacher education, or pre-service education for other positions 
in the field of vocational education. It refers to organized attempts 
to upgrade professional personnel in vocational education, such as those 
attempts funded under Section 553 of the Education Professions Development 
Act, but is not limited to those funded under Section 553* 

Project Director — The director of a personnel development project; the 
person in charge of the operation of the project. 

State Coordin ator of Personnel Development — The person designated by 
the state director of vocational education to coordinate or supervise the 
state personnel development program. The state coordinator is the person 
most directly responsible for state-level personnel development other 
than pre-service education. Several states use different terns for this 
person; for example, state director of personnel development and Part F 
coordinator. 

State Evaluation Team — The five-member group designated by the state 
coordinator of personnel development to meet and prepare an evaluation 
report on state personnel development. The team will use data collected 
with four survey forms and processed by the state coordinator of personnel 
development to prepare its report. 

Supervisor — The immediate supervisor of participants of personnel 
development projects. 

Vocational Education — The definition of vocational education used in 
the 1968 Amendments to the Vocational Education Act of 19^3* 
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GUIDELINES FOR DEVELOPING 
BEHAVIORAL OBJECTIVES 



A Model to Evaluate Inservice 
Personnel Development in 
Vocational -Technical Education 



The Center for Vocational and Technical Education 



The Ohio State University 
Columbus, Ohio 



These guidelines are designed to assist the state coordinator of 
personnel development and directors of personnel development projects 
in writing behavioral objectives for the personnel development program 
and projects. They or other sources dealing with behavioral or perfor- 
mance objectives, such as those listed on the last page, may be used to 
develop objectives. These guidelines, in other words, are offered as 
only one source of assistance in writing objectives. They present: 

(l) a brief definition of behavioral objectives, (2) guidelines to 
develop behavioral objectives, (3) examples of behavioral objectives, 
and (4) a suggestion to aid in writing difficult objectives. 

Definition of a Behavioral Objective 

A behavioral objective is a statement that describes what a person 
or group of people will be able to do after completing a particular kind 
of instruction. It specifies: (1) the instruction, (2) the behavior 

of behaviors that are to result from that instruction, and (3) the cri- 
terion level the behavior must reach. 1 Behavior is defined here as any 
human activity that can be observed by another and can be either measured 
or judged. 

Guidelines for the Development of Behavioral Objectives 

To facilitate writing behavioral objectives, it is best to divide 
each into its five components. An illustration of one behavioral objec- 
tive and the five components which apply to all behavioral objectives 
follow. 



By fiscal year 1975 > state coordinators of personnel 
development and project directors who use this evaluation 
modelmwill develop and use behavioral objectives, with the 
five components of each objective, for at least 50 percent 
of their objectives, as determined by state evaluation 





teams. 






(1) 


The person or persons who perform the desired behaviors 
state coordinators of personnel development and project 
tors who use this evaluation model) 


(e.g. 

direc 




(2) 


The desired behavior (e.g., will develop and use) 




to 


(3) 


The result or product of the behavior which will be evaluated 
(e.g., behavioral objectives, with the five components of a 
behavioral objective) 



•^Kibler, Robert J., Larry L. Barker, and David T. Miles. 
Objectives and Instruction . Boston, Mass.: Allyn and Bacon, 



Behavioral 
Inc., 1970. 
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(4) The relevant conditions under which the behavior is to occur 
(e.g., by fiscal year 1975) 

(5) The standard which will be used to judge the result or pro- 
duct of the behavior (e.g., for at least 50 percent of their 
objectives, as determined by state evaluation teams )2 

After writing behavioral objectives, check to make sure that each 
objective includes the five components. To prevent excessive detail in 
objectives for state programs, few objectives, perhaps 10 or less, should 
be written. 

The checklist and the three examples of behavioral objectives that 
follow may help persons writing objectives. The checklist can serve as 
a diagnostic tool to detect and correct errors in the behavioral objec- 
tives. Any negative answer indicates that an objective is not completely 
behavioral in terms of the definition presented here. 



CHECK LIST 



Criteria Yes No 



1. Does each behavioral objective emphasize a verb 
that requires action on the part of the student? 

2. Is each behavioral objective stated in terms of 
student performance (rather than teacher perfor- 
mance)? Does it describe what the learner will 
do when demonstrating this achievement of the 
objective? 

3. Is each behavioral objective stated so that it 
indicates terminal behavior (rather than subject 
matter to be covered during instruction)? 

4. Is each behavioral objective stated so that it 
includes only one learning outcome (rather than 
a combination of several outcomes ) ? 

5. Is there a sufficient number of behavioral 
objectives to adequately describe the desired 
achievement of the learners? 



2 Ibid. 
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Examples of Behavioral Objectives 

The following behavioral objectives are divided into their com- 
ponents. The number above each component refers to its numbered 
explanation on pages 1 and 2. 
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1. After inservice training or independent study, 



middle school teachers 



2 

will use 



at least two 



all elementary and 



career education 



concepts as a part of their curriculum. 



Division of Vocational Education. 



as judged by the State 



2. Every teacher at Metropolitan Technical Institute 



will present 



a five -minute lecture 



to nine of his peers on a topic in his 





2 


3 


k 


area 


and then privately view 


the lecture 


on video tape. 



He will determine the success of his presentation by completing 

a self -evaluation form. 

3. At a one-week seminar conducted by the Institute on Group Relations, 

1 

50 community college vocational education teachers within the state 



will acquire 



information about the culture of and probl sms encoun- 



tered by blacks and Mexican-Americans . | Attendance at every session 

5 

and a score of at least 70 percent on a test devised by the instruc- 
tors of the seminar will be accepted as successful completion of 
the seminar. I 

I 12 



Difficult Objectives 



Many times it is easier to write about behavioral objectives than 
to develop and use them in a practical situation. Kibler, Barker, and 
Miles offer the following advice for writing difficult objectives. 

Several types of extremely important objectives are 
difficult to measure and thus, difficult to specify in 
behavioral terms. As a matter of fact, it seems that the 
more significant an objective, the more difficult it is 
to measure. Examples of objectives which fall into the 
difficult-to-specify-and-measure category are those in 
the area of problem-solving, creativity, attitudes, and 
values. The only solution we see to this problem is for 
such objectives to be specified as clearly as possible 
and for the instructor to be as resourceful as he can 
in developing evaluative measures... 3 
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Suggested References for Writing 
Behavioral Objectives 



Kibler, Robert J.; Barker, Larry L., and Miles, David T. Behavioral 
Objectives and Instruction . Boston: Allyn and Bacon, Inc., 1970. 



Armstrong, Robert J. j Cornell, Terry D.j Kroner, Robert E., and Roberson, 

E. Wayne, eds. A Systematic Approach to Developing A Handbook Designed 
to Increase the Communication of Laymen and Educators . Tucson, Ariz. : 
Educational Innovators Press, Inc., 1968. ‘ 

Gronlund, Norman E. Stating Behavioral Objectives for Classroom Instruction . 
New York: The MacMillan Company, 1970. 

Hernandez, David E. Writing Behavioral Objectives . New York: Barnes and 
Noble, Inc., 1971. 

Mager, Robert F. Preparing Instructional Objectives . Palo Alto, Calif. : 
Fearon Publisher, 1962. 
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A Model to Evaluate Inservice 
Personnel Development in 
Vocational-Technical Education 



The Center for Vocational and Technical Education 
The Ohio State University 
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The purpose of this manual is to provide instructions for the distri- 
bution and collection of four survey forms and the selection of the state 
evaluation team (SET). The survey forms are: the state program coordinator 

survey (s-l), the project director survey (S-2), the participant survey 
(S-3), and the supervisor survey (s-4). The instructions are directed 
to the state coordinator of personnel development, but they may be used 
by anyone he or the state director of vocational education designates. 



Suggested Instructions for Distribution 
and Collection of Survey Forms 

1. Send the project director survey to directors of personnel development 

projects. 

a. Make a list of the personnel development projects operating 
during the year to be evaluated. 

b. Assign a number to each project. Assign 1 to the first project, 

2 to the second project , and so forth. 

c. Place the project numbers on the project director surveys next 
to the letter A in the block "For State Use." Place one number 
on each form. 

d. Check to make sure that each project director survey has the 
correct identifying number on it. 

e. Print the title of the project in the space provided at the 
top of the first page. 

f. Send the project director survey forms, with a cover letter, to 
the appropriate project directors. 

g. Request that the project directors send a one-page abstract of 
the project (without results) for use by the state evaluation 
team. This abstract can be one prepared previously. 

h. Have the project director survey forms and project abstracts 
returned to the state coordinator of personnel development. 

2. Obtain lists of participants and their supervisors. (This is suggested 

as a separate step, instead of as a part of instruction 1, to save time. ) 

a. Telephone each project director and request: (l) an alphabetical 

list of the names and addresses of participants who began the 
project, regardless of whether they completed it, and (2) the 
names and addresses of immedi ate supervisors of the participants. 

To simplify the selection of supervisors later, the name of each 
supervisor should appear next to the name(s) of participants 
who work for the supervisor. 
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b. If the project director does not have access to supervisors' 
names, request them from project participants. Be sure to 
emphasize that supervisors will not have access to the parti- 
cipant surveys, and that supervisors' names are requested only 
to facilitate the distribution of supervisor surveys. It is 
important to assure the participant that his responses are 
confidential. For future use of the evaluation model, instruct 
the project director to have the names of supervisors available. 

Complete the state program coordinator survey. 

Determine how many participants from each personnel development 
project will be sampled. 

a. Record the number of participants in each project. 

b. If the number is more than 4l, take 25 percent of that number. 

(Round off to the nearest whole number if the result is a 
fraction. One-half rounds off to the next highest whole number. ) 

c. If the number is 4l or under, always select 10 participants.. 

(if the project has 10 or less participants , use everyone in 
the project. ) 

d. The number obtained in either step 4b. or 4c. is the number of 
participants to be sampled from each project. An example of 
the participant selection procedures appears on page 6. 

Determine the specific participants to receive participant survey 
forms. 

a. If there are 10 or less participants in a project, use all 
participants. 

b. If there are 20 or more participants in a project, do the following: 

(1) Pick at random any name in the list. 

(2) Starting with the name just below it, count down until the 
fourth name is reached. 

(3) Circle this name. It is the name of the first participant 
selected. 

(4) Then start with the name just below the circled name and 
again count down until the fourth name is reached. 

( 5 ) Circle this name, 
selected. 



It is the name of the second participant 



3 




i 

> 

} 

| 



l 



L 

t * 

V 




(6) Count down to the next fourth name in the same manner and 
write it as the third participant selected. 

(7) Repeat this process until the required number of names is 
obtained. 

(8) When you reach the bottom of the list, continue at the top. 

(9) If you come to a name that has already been selected, select 
the name just below it and continue as before. 

c. If there are 11-19 participants in a project, use the same 
procedure as in step 5b. except that each name circled should 
be eliminated. Keep eliminating names until the names of 10 
participants remain. 

d. Use this selection procedure for each project. Examples follow 
on pages 6, 7> and 8. As stated in instruction 4, select 

10 participants for projects with 4l or less, and 25 percent of 
the participants for projects with 42 or more. 

6. Select only supervisors whose employees were selected in the 
participant sample obtained by using instruction 5 above. 

7* Assign numbers to the selected participants and supervisors. 

a. Prepare pages of 8£" x 11“ paper with three wide columns. The 
first column should contain the project titles, the second, the 
names and addresses of selected participants from each project, 
and the third, the names and addresses of the supervisors of 
selected participants. Leave space on the right side of each 
column for an identifying number. 

b. Instructions for assigning project numbers have been given (see 
instruction l). Place the correct identifying project number to 
the right of each project title in the first column. 

c. Use the space on the right side of the second column to assign 

a number to each participant. Assign 1 to the first participant, 
2 to the second participant, and so forth. 

d. Assign participant numbers across projects, so that if the 
number 40 is assigned to the last participant of project 3, 

4l is assigned to the first participant of project 4. No two 
participants should have the s«m<> number . 

e. Assign a number to each supervisor in the same manner. Assign 
1 to the first supervisor, 2 to the second supervisor, and so 
forth. Assign supervisor numbers across projects also. If a 
supervisor's name appears more than once, be sure to use the 
same number each time it appears. For example, if the name 

P. T. Jones is assigned number 2 initially and then reappears in 
the list, assign it number 2 again. 
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f. The lists of projects, participants, and supervisors should be 
kept on file. 

8. Assign identifying numbers to the participant surveys. 

a. Fill out the block at the top of the first page that says 
"For State Use. " 

(1) Next to A, write the project number. 

(2) Next toB, write the number of the participant's supervisor. 

(3) Next to C, write the participant's number. 

b. Print the title of the project, the agency involved, and the 
beginning and completion dates of the project on the "Project" 
line at the top of the first page. 

9* Prepare the supervisor surveys for distribution. 

a. If a supervisor has only one employee selected as a participant, 
no collation of the surveys is necessary.. 

b. If a supervisor has more than one employee selected as a partici- 
pant, make enough copies of the back , or second, page to insure 
that the supervisor has one copy for each of his participants. 

For example, if a supervisor is to rate four participants, make 
three copies. Attach these to the original supervisor survey 
form. 

10. Assign identifying numbers to the supervisor survey. 

a. At the top of the front of the page, under "For State Use," 
write the supervisor number next to B. 

b. At the top of each back page, under "For State Use," write 
the number of the participant's project next to A end write 
the participant number next to C. 

c. Also, at the top of each back page, print the name of the 
participant on the first line and the project title, the 
agency involved, and the beginning and completion dates on 
the second line. 

/ 

11. Mail the participant surveys and the supervisor surveys, with cover 
letters, to selected participants and supervisors. A follow-up 
letter or call may be used to assure a high percentage of returns. 

Be sure to include a stamped self-addressed envelope to those in 
the sample, to specify the exact project about which they are 
responding, and to specify exactly to whom the forms should be 
returned at a time determined by the state coordinator and his staff. 



Example of Determining the Number 
of Participants Selected from 
Each Project 

Suppose state Y has a total of 166 participants in five projects. 
The procedure of determining the number in each project to be selected 
is illustrated in the table below. 



Project 


Number 


Number Selected 


1 . 


32 


10 (10 selected because there 

are fewer than 42 participants) 


2. 


6 


6 (everyone used because there 
are fewer than 10 participants) 


3* 


42 


11 ( 2 % of 42, rounded off) 


4. 


23 


10 (10 selected because there 

are fewer than 42 participants) 


5. 


63 


16 ( 2 % of 63 , rounded off) 



Example of Selecting Participants From 
Each Project 

On the next two pages are examples of participant lists from projects, 
one with l6 participants and another with 21. Ten participants should 
be selected from each project because they both have less than 42 parti- 
cipants. For the project with l6 participants, names axe eliminated , as 
illustrated, until 10 remain. For the project with 21 participants, 
names are selected , as illustrated, until the necessary number of 10 is 
reached. 



Example 



Selection of 10 Participants 
from a Project with 16 Participants 



Project 4 



T 








3. 




Adams, Carolyn 




4. 




Brinkman, Vera 




1 . 




Crawford, William 


2. 




Dram, Henry 




3. 




Hanson, Burt 




4. 




Holson, Teresa 




1 . 




Karl, Thomas 




2. 




Munn, Nancy 




3. 




Neilson, Anne 




4. 


(Start) 


Paulson, William 


1 . 


1 . 


Pliny, Curtis 




2. 


2. 


Plotter, Allan 




3. 


3. 


Richter, Warren 


4.* 


4. 


Samson, Richard 










4. 


1 . 


Sawyer, Kenneth 


1 . 


2. 


Talbert, Diane 





Start : Random selection 

of any name on 
the list 



2nd name eliminated 
6th name eliminated 



3rd name eliminated 



4th name eliminated 



1st name eliminated 
5th name eliminated 



Since this name, Richard Samson, was already elminated, 
the name ju 3 t below it, Kenneth Sawyer, is now eliminated. 
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Example: 



f 

4. 



1 . 



(Start) 



2 . 



1 . 



3- 

4. 



1 . 



2 . 



3- 

4. 



2 . 



1 . 



3- 

4. 



1. 



2 . 



3. 

4. 



2 . 



1. 



3- 



2 . 



4. 



1. 



3. 

4. 



2 . 



1. 



3- 

4. 



1. 



2 . 



3. 

4. 



2 . 



1 . 



3. 



2 . 



4. 



Selection of 10 Participants 
from a Project with 21 Participants 



Project 1 

j Barter, Samuel 
Barton, Harriet 
Corbett, Donald 
Dixon, Carl 
Earl, Linda 
Granger, Daniel 
Hammond, Elizabeth 
Jacob, John 
Lerner, Robert 
McMahon, Frank 
Myer, Mary 
Neely, William 
Pratt, John'l 
Preston, Cindy 
Riley, Eileen 
Rinley, Gail 
Slade, Karen 
Sutter, Robert 
Travis, David 
Voss, Kenneth 
Walters, Alex 



Start : Random selection 
of any name in 
the list 

5th name chosen 



6th name chosen 
1st name chosen 



7th name chosen 
2nd name chosen 



8th name chosen 
3rd name chosen 



9th name chosen 
4th name chosen 



10th name chosen 
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Illustration of a List of Proj ects, Selected Participants. 
and Supervisors . with Identifying Numbers 
( Only a Partial List ) 



TITLE OF PROJECT 


NO. 


PARTICIPANT 


NO. 




SUPERVISOR 


NO. j 


Instruction of the Handicapped 


1 


James Jones 


1 




Philip Weber, Prin. 


1 ; 


in Vocational Education 




115 Wisconsin St. 






Novak High School 








Silver Springs 96115 






200 Eureka Park 
Silver Springs 96112 






1 


Tom Delfino 


2 




Samuel Ceccari 


2 






Route 4, Box 400 






Prin., Ponderosa H.S. 


’ 






Woodland 95402 






East Horse Road 
Woodland 95400 






1 


David Tolbert 


3 




Mark Delpond 


3 






15 Monte Drive 






Vice Prin. 








Clairsville 94321 






Woodfield High School 
Clairsville 94321 






1 


Burt Adams 


4 




Samuel Ceccari 


2 






P. 0. Box 75 






Prin., Ponderosa H.S. 








Fayeville 94415 






East Horse Road 
Woodland 95400 






1 


Jerry Osborn 


5 




Mark Mercer, Dir. 


4 i 






1620 Eastmoor Avenue 






Voc-Ed, Madison 








Dayton 95234 






County Dept, of Ed. 
Dayton 95233 






1 


Steven Swings 


6 




Warren Lamoint 


5 






22 Bubble Street 






Prin., Marysville H.S, 




; 




Marysville 95004 






Layfatee Road 
Marysville 95005 




i 


1 


Jackie Hess 


7 




George Brummley 


6 ; 






1313 Joanna Avenue 






Prin., Central H.S. 


i 






Marion 96332 






4l6 Fourth Street 
Marion 96332 


i 


i 

■ Automobile Repair Workshop 


2 


P. L. Fry 


8 




Jerry Stephens, Prin. 


7 ; 






107 Louis Drive 






Hartley High School 


! 


j 




Ocala 98014 






601 Main Street 


! 


f *■ 










Ocala 96020 




r 


2 


Stephanie Woodland 


9 




James Mariner, Prin. 


8 


V 

t 




380 Wood Avenue 






Alder High School 




b 

i- 




Orange 95441 






Amity Road 




c 

i*' 

e 

F 










Orange 95445 






2 


John Hanson 


10 




Ron Ebright, Prin. 


9 






4797 Good Avenue 






South High School 




0 




Los Angeles 93221 






Crosby Boulevard 




ERLC 


1 


1 „ 






Los Angeles 93224 





SH 
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Suggested Instructions for Selection 
of the State Evaluation Team 

The state coordinator should select five persons, including a 
chairman, to serve on a state evaluation team (SET). He should schedule 
a two- or three- day meeting for the SET members and himself. 

The state coordinator should choose no more than one person for 
the SET from each of the following groups: 

1. District or secondary administrators or faculty members from 
a local education agency 

2. The state advisory council for vocational education 

3. Faculty members in vocational-technical education from a 
university or college within the state 

4. Faculty members (who specialize in research or evaluation) 
from a university or college within the state 

5. The division of research or evaluation within the state department 
of education 

6. Administrators or faculty members of vocational-technical 
education from the community college or technical institute 
within the state 
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The purpose of this manual is to provide the state coordinator of 
personnel development, or those delegated by him, with instructions to 
prepare data for thj use of members of the state evaluation team (SET). 
The instructions include specific steps and a proposed format for data 
used in the evaluation. Two appendices follow the text of this manual. 
These are: (1) tables which serve as the format for hand-processing 

data, and (2) a computer program and instructions to use it in process- 
ing data. 

The initial data used by the SET will come from the state program 
coordinator survey (S-l), the project director survey (S-2), the par- 
ticipant survey (S-3), and the supervisor survey (S-4). Instructions 
to collect data with the survey forms appear in the Manual of Instruc- 
tions to collect data with the survey forms appear in the Manual of 
Instructions for the Distribution and Collection of Survey Forms and 
Selection of the State Evaluation Team (M-l). The instructions in this 
manual, M-2, are directed to the state coordinator of personnel devel- 
opment, but they may be used by anyone he or the state director of voca- 
tional education designates. 

The state coordinator should use the instructions which follow to 
prepare data for the SET. He will arrange for each member of the team 
a program data folder with information about the state personnel devel- 
opment program and project data folders with data on individual projects 
of the state program. Instructions to prepare the program data folder 
and project data folders follow. 
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Instructions to Prepare Program and Project Data Folders 



Allow sufficient time for project directors, participants, and 
supervisors to return the surveys sent to them. As indicated 
above, instructions for distributing and collecting the survey 
forms are in M-l. 

When the forms have been returned, place them in three separate 
groups or stacks according to the persons who completed them: 
project directors, participants, and supervisors. 

Place the survey forms within each of the three stacks in numer- 
ical order by using the project numbers for project director 
surveys, participant numbers for participant surveys, and super- 
visor numbers for supervisor surveys . The project number appears 
next to the letter A in the top right corner of the survey forms, 
the supervisor number next to B and the participant number next 
to 0 . 

Prepare a program data folder for each SET member. 

a. Obtain five folders. 

b. Label each: "Program Data Folder." 

Prepare project data folders for project data. 

a. Select the needed number of folders. This number is equal to 
the number of projects times five. For example, if there are 
four projects, 20 project data folders should be prepared 

(4x5- 20). 

b. Label the project data folders. Five should be labeled "pro- 
ject 1," five "project 2," and so forth until five project 
folders are labeled for each project. 

Decide whether to break out program data by size and duration of 
projects. If so, choose categories and include them in the com- 
puter program or write them in the program tables to be used for 
hand analysis. (See instructions 8f and 8h for placing data in 
categories . ) 

% 

Decide whether or not to use a computer. The main criteria to 
consider in the decision are the number of projects and the total 
amount of data. If a computer is used, follow the instructions 
in Appendix 2 and then go to instruction 9* If not, and hand 
analysis is necessary, use instruction number 8 which follows. 

NOTE: Table 5a (instruction 8g) should be 

completed by hand in either case . 
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8. Prepare Tables 1-13 (including 5®) with data for both projects and 

the program. The instructions they follow are for hand analysis. 

They assume a small number of projects and little data. 

a. Make sufficient copies of Tables 1-13 from Appendix 1 to equal 
the number of personnel development projects plus one. For 
example, if there are four projects, make five copies of each 
table. 

b. Label one copy of each of the Tables 1-13 (and 5a) as " project 1 " 
Also label one copy of each of the Tables 1-13 for each of the 
other projects. For example , if there are four projects, there 
should be copies of Tables 1-13 labeled, project 1, copies label- 
ed project 2, copies labeled project 3» and copies labeled 
project 4. The first row of each project table should also be 
labeled with the project number, for example, project 1 with 
"project 1," project 2, with "project 2," and so forth. 

c. There should be one copy of each of the 13 tables remaining. 

Label the remaining tables as " program" tables. Label the first 
row of the program tables "all projects . " 

d. List the size and duration categories to be used in program 
Tables 1-4 and 7-11. There should be nine program tables with 
these categories listed (1-4 and 7-H) • 

e. Decide whether to use numbers or percentages in the tables, or 
both. 

f. Take the first participant survey from the stack of participant 
surveys (instruction 3)* Using light pencil tally marks, trans- 
fer data from the survey form to Tables 1-6 (excluding Table 5a) 
for project land program tables 1-6. Use only the top row for 
tallies in the project tables; that is, do not use the rows for 
size and duration. In the program tables make tally marks for 
the top row and the appropriate rows for size and duration. 

Each of the program tables, therefore, will have tally marks 

in three rows: (1) the row labeled "all projects," (2) one of 

the four rows for project size, and (3) one of the four rows 
for project duration. 

If a participant does not respond, make a tally mark in 
the column "No response." For every tally mark made, also 
make a tally mark in the last column, "No. of participants," 
so that a count can be kept of the participants in each row. 

Make a mark in the last column even if the "No response" 
column is the one that is marked. See the example on the 
following pages. 
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Pro.lect Table : 



TABLE 1, Project d- 

Participant Reaction to "How worthwhile was the project to you?": 
All Projects, by Project Size and by Project Duration 
(Expressed in Numbers and/or Percentages) 



Projects 

1 


How worthwhile was the project for you? 


No. of 
partici- 
pants 


Worthwhile 


Not worthwhile 


No 

response 


Completely 


Partially 


Partially 


Completely 






l* 










5 roject size 
by number of 
>articipants ) 


























































Project 
luration 
'in days) 



























































I 



w 



w 
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Program Table : 



TABLE 1) Pt~o<^r‘ (Atn\s 

Participant Reaction to "How worthwhile was the project to you?": 
All Projects, by Size and by Duration 
(Expressed in Numbers and/or Percentages) 



Projects 


How worthwhile was the project for you? 


No. of 
partici- 
pants 


Worthwhile 


Not worthwhile 


No 

response 


Completely 


Partially 


Partially 


Completely 


; f\W S 




t* 










roject size 
by number of 
participants ) 


I-*© 


















l* 








j* 


£l - to© 














over \00 














project 
duration 
in days) 

i 


7 


.. 












6 - IH 














ir* at 












1* 


OV»er 















* The participant checked "partially worthwhile" (see page 4). 
Tally marks should be made under the "partially worthwhile " 
column four times: once for project 1, once in the first row of 

the program table, once in the row for a size of 50 participants 
of the program table , and once in the row for a duration of 
17 days of the program table. In addition, tally marks should be 
made under the "No. of participants" column four times, in the 
same four rows described above. (Remember, if the participant 
does not respond, tally marks should be made under the "No 
response" column and the "No. of participants" column.) 
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Continue in the same manner with participant forms until 
data from project 1 participants have been tallied on project 1 
tables 1-6 (excluding table 5a) and program tables 1-6. Tally- 
participant data on project tables 1-6 and program tables 1-6 
for each of the remaining projects. Be sure to use the same 
program tables for data from all projects. 

g. Take the project director survey for project 1 from the stack 
of project director surveys (instruction 3) • Transfer data 
from the table in item number 4, page 2, to table 5a for pro- 
ject 1. Repeat this process for table 5a for project 2, table 
5a for project 3, and so forth. When all project director sur- 
veys have been tallied, total them and erase the tally marks. 
Table 5a will be prepared in the same manner in states using 
the computer program . 

h. Take the first supervisor survey form from the stack of super- 

visor surveys (instruction 3). Using light pencil tally marks, 
transfer data from the supervisor survey forms to tables 7-13 
for project 1 and for program tables 7-13. (Tables 12 and 13 
are special cases. Table 12 will require the use of both sides 
of the supervisor survey and neither 12 nor 13 have rows for 
size and duration . ) In project tables 7-11, use only the top 
row for tallies; that is, do not use the rows for size and 
duration. In program tables 7-11, make tally marks for the 
top row and the appropriate rows for size and duration. The 
program tables 7-11, therefore, will have tally marks in three 
rows: (1) the row labeled "all projects;" (2) one of the four 

rows for project size; and (3) one of the four rows for pro- 
ject duration. Continue in the same manner with supervisor 
forms until data from project 1 supervisors have been tallied 
on project tables 7-13 and program tables 7-13 • Tally super- 
visor data on project tables 7-13 an d program tables 7-13 for 
each of the remaining projects. Be sure to use the same pro - 
gram tables for data from all projects. 

i. Group the pro j ect tables with the same table number. That is, 
put all table l's together, all table 2's together, and so 
forth . 

j. For each of the project tables, total the tally marks in the 
cells of the top row and write that total in the cell. For 
the program tables, total the tally marks in cells of all 
rows and write the totals in the cells. 

k. If the tables are to be completed with percentages as well 
as numbers, or in place of numbers, compute the percentages 
which should appear in each cell. 
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L. Group the project tables by project number . That is, put 
■ \ together tables 1-13 ( including 5a) with data for each of 

the projects. For example, if there are four projects, there 
should be four sets of tables 1-13, one for each project. 

m. Make five copies of each set of project tables for members 

of the SET. That is, if there are four projects, each of the 
five SET members should receive copies of tables 1-13 ( in- 
cluding 5a) for projects 1, 2, 3, and 4. 

n. Prepare five copies of the program tables, one for each 
member of the SET. 

Prepare program folders for each of the .five members of the SET. 

a. Place a program description sheet in each of the program data 
folders . The description sheet should include the following 
information. 

—Number of personnel development projects in the state 
— Number of projects sampled for the evaluation 

— Number of participants of personnel development projects in 
the state 

— Number of participants sampled for the evaluation 
—Number of participants responding 

—Number of participants in the sample who did not complete 
projects (This number comes from participant surveys and 
can be obtained by hand analysis.) 

— Number of supervisors to whom surveys were sent 
—Number of supervisors responding 

—Number of responding participants whose supervisors did not 
respond 

— Number of responding supervisors whose employees (partici- 
pants) did not respond 

b. Place one copy of the completed state coordinator survey (S-l) 
in each of the five program folders. The completed form should 
follow the program description sheet . 

c. Place copies of program tables 1-13 (including 5a) prepared 
either by the computer or hand analysis, in the program data 
folder . These should follow the state program coordinator 



survey. 
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10. Prepare proj ect data folders for each of the five members of the 
SET. 



a. Place a copy of the project description sheet for each project 
in each project data folder. Since each SET member will re- 
ceive one project data folder for each project, each will 
receive as many project description sheets as there are pro- 
jects. The sheet should include the information which follows. 
— Number of participants in the project 

— Number of participants sampled 
—Numbers of participants responding 

--Number of participants in the sample who did not complete 
the project (This number comes from the participant survey. 

It can be obtained by hand analysis . ) 

— Number of supervisors to whom surveys were sent 
— Number of supervisors responding 

— Number of responding partic ipants whose supervisors did not 
respond 

— Number of responding supervisors whose employers ( partici - 
pants ) did not respond 

b. Place a copy of the completed project director survey for the 
appropriate projects in the project data folders for each of 
the five SET members. For example, if there are four projects, 
each SET member should receive in his project data folders, 
four project director surveys, one each for projects 1, 2, 3, 
and 4. The survey forms should follow the project description 
sheets . 

c. Place copies of project table s in the project data folders. 

Each SET member should have tables 1-13 (and 5a) for each 
project. For example, if there are four projects, tables 1- 
13 (including 5a) for project 1 are in one folder, tables 1- 
13 for project 2 are in another folder, and so forth. The 
tables should follow the project director survey in the folders. 

d. Tear off pages 3 and 4 of the participant surveys and make 
copies. Place the copies in the corrent project data folders. 
(These pages have provisions for comments.) Pages 3 and 4 
should follow the project tables . 

e. Place copies of the supervisor surveys in the corrent project 
data folders so that the back or second page is up. (This 
page has space for comments.) 
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11. Prepare materials for the SET members. 

a. Before the SET meeting, send members the materials which 

follow: 

— The Overview of the evaluation model 

— The Manual of Instructions for the State Evaluation Team 
Meeting (M-3) 

— Project abstracts for each project (short summaries which 
include the purpose(s) of the project, the duration, and 
other descriptive data). 

b. Prepare to distribute materials to members of the SET at the 

SET meeting. 

— Program data folders 

— Project data folders (Each SET member should receive a pro - 
ject data folder with tables 1-13 for each project.) 
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No. of 
partici- 
pants 




















| How well organized was the project? 


No 

response 




















Not organized 


| Completely 




















| Partially 




















Organized 

1 


Partially 




















Completely 




















Projects 




















Project size 
(by number 
of partici- 
pants) 


Project 
duration 
(in days) 
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Participant Ratings of Instructional 
Activities 



asuodsaj: on 














papaau q.nq 
pssn q.o n 














pasn q.oM 














Not 

worthwhile 


jCiacj-apdoioo 




























Worthwhile 
















jCxaq.aiduioo 














Instructional 

activities 
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b. Audiovisual aids 


c* Classroom discussions 


d. Textbooks and reading materials 


e. Actual or simulated work 

experiences related to the job 


f. Others (Please specify) 



TABLE 5a 



The Extent to Which Instructional Activities Are Used 
or Not Used in Personnel Development Projects 
(Expressed in Numbers and/or Percentages) 



Instructional activities 


Used 


Not 

used 


Much 


Little 


Guest lecturers and speakers 








Audiovisual aids 








Class discussion 








Textbooks and reading materials 








Actual or simulated work exper- 
iences related to the job 








Others 
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participants) 


Project 

duration 


(in days) 
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TABLE 8 



Supervisor Reaction to "Has there been a noticeable improvement in the job 
performance of participants after participating in the project?": 

All Projects, by Size and by Duration 
(Expressed in Numbers and/or Percentages) 



Projects 


Has there been a noticeable improve- 
ment in the job performance of 
participants . . . ? 


No. of 
super- 
visors 


Mich 


Some 


No 


No 

response 














Project 
size (by 
number 
of par- 
tici- 
pants) 


















































Project 
duration 
(in days) 
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TABLE 9 



Supervisor Reaction to "Would you encourage other employees of your 
agency to participate in a similar project?": 

All Projects, by Size and by Duration 
(Expressed in Numbers and/or Percentages) 



Projects 


Would you encourage other 
employees of your agency to 
participate in a similar 
project? 


No. of 
super- 
visors 


Yes 


No 


No 

response 












Project 
size (by 
number 
of par- 
tici- 
pants) 










































Project 
duration 
(in days) 











































TABLE 10 



Supervisor Reaction to "Has participant shared his experiences 
with you, his immediate supervisor?": 

All Projects, by Size and by Duration 
(Expressed in Numbers and/or Percentages) 



Projects 


Has participant shared his 
experiences with you, his 
immediate supervisor? 


No. of 
super- 
visors 


Yes 


No 


No 

response 












Pro j ect 
size (by 
number 
of par- 
tici- 
pants) 










































Project 
duration 
(in days) 











































TABLE 11 



Supervisor Reaction to "Has participant shared experiences with 
other employees with whom he works?": 

All Projects, by Size and by Duration 
(Expressed in Numbers and/or Percentages) 



Projects 


Has participant shared experiences 
with other employees with whom he 
works? 


No. of 
super- 
visors 


Yes 


No 


Do not 
know 


No 

response 














Project 
size (by 
number 
of par- 
tici- 
pants 


















































Project 
duration 
(in days) 
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Supervisor Reactions to "How well did the project apply to participant's work?" and "Has there 
been a noticeable improvement in the job performance of a participant after 
participating in the project?" by Size of School District 
in Pupils and Educational Agency of Supervisors 
(Expressed in Numbers and/or Percentages) 



No. of 
super- 
visors 
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How well did the project apply to 
participant's work? 




























1 Did not apply 1 


$ 

O 


























I 

* 


























5 

1 


| 


























s 

Q 


$ 

o 


























Size of School 
District and 
Educational Agency 


Less than 10,000 


o 

CM 

H 

8 

« 

S 


o 

ir> 

H 

8 

• 

O 

CM 


O 

ir> 

a 

€0 

,c 

•p 

<u 

Jh 

* 


College or 
University 


Conan College or 
Technical Inst. 


Intermediate School 
Unit (Regional, 
i Area or County Unit 


Local School 
District 


State Department 
of Education 








aaqqO 


siTdnd-^OTJ^STa 
loonos jo azfs 


sJosjAJddns 
jo Ji oua3v xeucp'j.Bonpa 



0) 

S^>i 

0 <U H 
<D H 

^UlS 

1 I I 



* * I 

PC * * 



TABLE 13 



Provisions Made by Supervisors for 
Participants to Share Experiences 
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Provisions for Sharing 
Experiences 


Percentage of Super- 
visors Making Provisions 


Group seminars 




Newsletters 




Informal discussions 




Others 
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Appendix 2 



Computer Program and Instructions 
For Its Use in Processing Data 



EPDA - B60: PROGRAM EP02 



The program reads records from the following files: 
Project Director's Survey 
Participant Survey 



Prom the information stored on these folies, EP02 generates the 
following six tables: 



TABLE 1: Participant Reaction to "How worthwhile was 

the project to you?" 

TABLE 2: Participant Reaction to "How well did the project 

apply to your work." 

TABLE 3: Participant Reaction to "How well organized was 

the project?" 

TABLE k: Participant Reaction to "Would you recommend this 

project to someone else?" 

TABLE 5: Participant Ratings of Instructional Activities 



TABLE 6: Participant Reaction to "How worthwhile was the 

project for your?" and "How well did the project 
apply to your work?" 



Tables 1 through b are broken down with respect to project size 
and duration and Table 6 is broken down with respect to age, sex, and 
race. All answers are given in percentages except for the last column 
in Tables 1, 2, 3, U, and 6. 

The program accepts one control card which enables the user to 
run the program for one project specifically or for all projects. The 
control card also contains a series of variables which allows the user 
to break down the input data with respect to different project sizes 
and durations. 

While processing, the program checks all input data for values 
that are not allowed and prints out error messages if errors are found. 
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System: EPDA Program: EPOU 



This program reads records from the following files: 

Project Director's Survey 

Supervisor Survey 

Supervisor Participant's Survey 

From the information stored on these files, EPOU generates the 
following seven tables: 

TABLE 7: Supervisor Reaction to "How well did the project 

apply to participant’s work?" 

TABLE 8: Supervisor Reaction to "Has there been a noticeable 

improvement in the job performance of participants 
after participating in the project?" 

TABLE 9: Supervisor Reaction to "Wbuld you encourage other 

employees of your agency to participate in a 
similar project?" 

TABLE 10: Supervisor Reaction to "Has participant shared his 
experiences with you, his immediate supervisor?" 

TABLE 11: Supervisor Reaction to "Has participant shared 

experiences with other employees with whom he works?" 

TABLE 12: Supervisor Reaction to "How well did the project apply 
to participant's work?" and "Has there been a notice- 
able improvement in the job performance of a partici- 
pant after participating in the project?" 

TABLE 13: Provisions made by Supervisors for Participants 
to Share Experiences 

Tables 7 through 11 are broken down with respect to project size 
and duration and Table 12 is broken down with respect to size of school 
district in pupils and educational agency of supervisors . All answers 
are given in percentages except for the last column in Tables 7, 8, 9» 

10, 11, and 12. 

The program accepts one control card which enables the user to run 
the program for one specific project or for all projects. The control 
card also contains a series of variables which allows the user to breed: 
down the input data with respect to different project sizes and dura- 
tions . 

While processing, the program checks all input data for values that 
are not allowed and prints out error messages if errors are found. 
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MANUAL OF INSTRUCTIONS FOR THE STATE 
EVALUATION TEAM MEETING 



A ffodel to Evaluate In service 
Personnel Development in 
Vocational-Technical Education 



The Center for Vocational and Technical Education 
The Ohio State University 
Columbus, Ohio 



The purpose of this manual is to assist members of the state eval- 
uation team (SET) to evaluate inservice personnel development in voca- 
tional-technical education in the state. The man ual provides: (l) guide- 

lines for the SET meeting and (2) guidelines for interpreting processed 
data. The guidelines only suggest direction and need not be followed 
literally. Members of the SET should adapt them to the immediate situation 
in which they are working. The manual also describes the responsibilities 
of the state coordinator of personnel development during the SET meeting 
(see Appendix). 

The SET is a five -member group selected by the state coordinator of 
personnel development, (instructions to select the SET are in M-l. ) The 
purposes of the state evaluation team are: (l) to interpret processed 

data in this manual, (2) to draw conclusions about the state program and 
projects from those data, (3) to make recommendations about program and 
projects , and ( 4 ) to incorporate the team's work into a final evaluation 
report for the state director of vocational education. 

The chairman of the SET may be appointed by the state coordinator of 
personnel development. (This instruction assumes that the state director 
of vocational education has assigned responsibility for the evaluation to 
the state coordinator. ) The chairman should be responsible for conducting 
the meeting and insuring that an evaluation report is delivered to the 
state director. Guidelines for the meeting and for interpreting data 
follow. 
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Guidelines for the SET Meeting 



1. The state coordinator should open the meeting with a brief intro- 
duction of the purpose of the meeting and the personnel development 
activities in the state. (See Appendix.) 

2. After the introduction, the state coordinator should be present only 
when he is needed, to supply information or for some other specific 
reason. The state coordinator should not be a voting member of the 
SET; he should be an ex officio member only. 

Team members may want an orientation period in which they introduce 
themselves and discuss their areas of interest. 

After the introduction, members should review briefly the materials 
they will use during the meeting. Those materials are shown below. 
(SET members will most often use this manual, M-3 , and processed 
data in the program data folder and project data folders.) 

a. Materials received from the state coordinator before the meeting: 

(1) Overview. General description of the entire evaluation 
model. 

(2) Manual of Instruction for the State Evaluation Team Meeting 
(M-3, this manual). Guidelines for the meeting of the state 
evaluation team and the interpretation of processed data. 

(3) Project Abstracts . Short summaries which include the 
purpose (s) of projects, their duration, and other descriptive 
data. 

b. Materials supplied by the state coordinator at the SET meeting. 

(1) Four data collection instruments: 

— State program coordinator survey (s-l) 

—Project director survey (s-2) 

— Participant survey (s-3) 

— Supervisor survey (s-4) 

(2) Program data folder 

(3) Project data folders (for each project) 

(4) Additional data the state coordinator wants to provide 
SET members. 

5. Team members should scan the program and the project data folders to 
obtain an overview of information in them. 
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6. Team members should read through the Guidelines for Interpreting 
Processed Data , which begins on page 4 of this manual. They should 
then use the guidelines to interpret data in the program and project 
data folders. Criteria have not been established for making judgments 
about the data; the team should develop its own criteria for judging 
data. 
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7> The team should incorporate its findings and conclusions into an 

evaluation report. The chairman, as stated above, should be respon- 
sible for insuring that the state director of vocational education 
receives a copy of the evaluation report. A suggested outline for 
the report follows. 

a. Introduction 

(1) Purpose of the report 

(2) Background information 

(3) Brief summary of what is to be evaluated 

b. Main body of the report 

(1) Descriptive data related to personnel development 

(2) Conclusions about personnel development 

(3) Cautions fbr use of the data 

(4) Recommendations 

c. Summary statement 



d. Appendices 

8. The state coordinator should supply clerical assistance needed by 
the team to prepare the final evaluation report. 



9. The chairman should make sure that the final copy of the report is 
ready and submit it to the state coordinator who, in turn, should 
submit it to the state director of vocational education. 
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Guidelines for Interpreting Processed Data 



This section of the manual provides guidelines, mainly in the form 
of questions, to assist members of the SET in interpreting data in the 
program data folder and project data folders. The questions in this 
section are only suggestive; they are intended to stimulate thought 
about the data and to suggest possible approaches for interpretation of 
the data. Ultimately, team members must use their own professional 
judgment to interpret the data used to prepare the evaluation report . 

The program data folder includes: (l) a program description sheet, 

(2) the completed state program coordinator survey (S-l), and (3) program 
tables that summarize opinions of project participants and their super- 
visors about all projects. The description sheet contains sampling infor- 
mation about state projects , participants of projects , and their super- 
visors. Categories of data in the state program coordinator survey (S-l) 
are: I. Determination of Personnel Development Needs, II. Needs and 

Priorities, III. Procedures for Selecting Projects, and IV. Objectives 
of Personnel Development Program. The opinions in the program tables are 
a summation of the opinions about projects; participants and supervisors 
did not give opinions about the entire program. 

The project data folders contain: (l) project description sheets, 

(2) completed project director surveys that provide a statement of project 
objectives and evidence of the attainment of those objectives, and 

(3) project tables that summarize opinions of project participants and 
their supervisors about the project. 

The questions that follow are divided into three groups. The first 
group relates to data about needs , selection procedures . and objectives. 
These data are in the state program coordinator survey (S-l) of the 
program data folder and the project director surveys (S-2) of the project 
data folders. The second group of questions relates to opinions of 
project participants and their supervisors . These data are in the program 
tables of the program data folder and the project tables and comments of 
the project data folders. The third group of questions relates to the 
data of the first two groups and requires judgments on the attainment of 
objectives . 



Heeds , Selection Procedures . and Objectives 

1. Needs (Refer to Sections I and II, pp. 1-3 of the state program 

coordinator survey in the program data folder. Questions a. through 
e. refer to Section I, and f. through i. to Section II.) 

a. What methods were used to determine the personnel development needs? 

b. What method was rated as most important? 
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c. How many persons were consulted? 

d. In your opinion, were there other means which might have been 
used to determine personnel development needs? 

e. Was the process used to determine needs consistent with the 
procedures set forth in the state plan for vocational education? 

f. What were the most important criteria for setting the priority 
of needs? 

g. In your opinion, could other criteria be considered in developing 
the priority of needs? 

h. Is the list of priorities consistent with the objectives for 
personnel development in the state plan? 

i. What were the major constraints which caused priorities to be 
lowered? 

j. What conclusions can be drawn from the data? 

2. Selection Procedures (Refer to Section III, pp. 4 and 5 of the state 

program coordinator survey in the program data folders. Questions 

a. through d. refer to Section IIIA, questions e. and f. to Section 

IIIB, and g. through i. to Section IIIC.) 

a. From which agencies were proposals submitted? 

b. What type of agency submitted the largest percentage of proposals? 

c. What type of agency had the largest number funded? 

d. Are there other types of agencies from which proposals might 
have been solicited? 

e. In your opinions were the procedures used to select proposals 
open and flexible? 

f. Might others have been used? 

g. Which criteria were most important in screening proposals for 
personnel development projects? 

h. In your opinion, are there additional criteria which might be 
used to screen proposals? 

• 

i. Was the screening process consistent with state objectives? 

(See Section IV of the state program coordinator survey in the 
program data folder.) 

j. What conclusions can be drawn? 
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3. Program Objectives (Refer to Section IV, pp. 5 and 6 of the state 

program coordinator survey in the program data folder. ) 

a. Are program objectives stated as behavioral objectives? (If 
you wish, you may request Guidelines for Developing Behavioral 
Objectives from the state coordinator as an aid. You may also 
use any other reference available to you. ) 

b. If not stated as behavioral objectives, are program objectives 
stated in specific terms? How can the program objectives be 
improved? 

c. Are program objectives consistent with the state needs and 
priorities for personnel development? (See Sections I and II 
of the state program coordinator survey in the program data 
folder. ) 

d. Does the evidence supplied for the attainment of program objectives 
seem adequate to render a judgment? (This evidence, along with 
the opinions of project participants and their supervisors, should 
be used in a later section of these guidelines to determine the 
extent to which objectives were attained. ) 

4. Project Objectives (Refer to pp. 3 and 4 of the project director surveys 

in the project data folders. ) 

a. Are project objectives stated as behavioral objectives? (Again, 
use any reference as an aid.) 

b. If not stated as behavioral objectives, are project objectives 
stated in specific terms? How can the project objectives be 
improved? 

c. Does the evidence supplied for the attainment of project objectives 
seem adequate to render a judgment? (This evidence, along with 
the opinions of project participants and their supervisors, should 
be used in a later section of these guidelines to detezmine the 
extent of attainment of objectives.) 



Participant and Supervisor Opinions 

1. Opinions about the program (Refer to the tables in the program folder.) 
Keep in mind that the opinions in the program tables are simply a 
summation of opinions about individual projects. No participant or 
supervisor was asked his opinion about the entire state program. Also, 
projects were not all sampled in the same manner. Since there was a 
minimum sampling number of 10, small projects may have the same or 
nearly the same representation as large projects. For example, a 
project of 10 participants has all 10 sampled, whereas a project of 
40 has only 25 % sampled, or 10 again. In interpreting the opinions 
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about the program, check to see if some projects have more influence 
than they should. A caution could be included in any conclusions 
made. (The breakdown by project size might also be helpful in 
interpreting these data, since small projects are analyzed separately. ) 
Finally, be careful about making conclusions where there is a high 
number of no responses or a low number of participants. 

a. Participant opinions 

(1) What conclusions can be drawn from the data in Tables 1-4? 

(it may be helpful to compare the similarity of responses 
in Tables 1 and 2. ) Make conclusions about all projects 
and about projects by size and duration. 

(2) What conclusions can be drawn from the data in Table 5? 

Be sure to compare this table with Table 5a completed by 
project directors. 

(3) What conclusions can be drawn about sex, age, or race from 
the data presented in Table 6? 

b. Supervisor opinions 

(1) What conclusions can be drawn from the data in each of the 
Tables 7-H? Make conclusions about all projects and about 
projects by size and duration. 

(2) What conclusions can be drawn from data presented in Table 12? 

c. Are there any ove rall conclusions, or recommendations, that can 
be made about the program as a result of reviewing the opinions? 

Opinions about the projects (Refer to tables and comments in the 
project data folders. ) Each project should be judged separately, 
so answer the questions below for each project. Only the top row 
of Tables 1-4 and 7-11 are relevant to projects. Be careful about 
making conclusions where there is a high number of no responses or 
a low number of participants. 

a. Participant opinions 

(1) What conclusions can be drawn from the data in Tables 1-4 
and the participant comments? 

(2) What conclusions can be drawn from the data in Table 5? Be 
sure to compare this table with Table 5a completed by the 
project director. 

(3) What conclusions can be drawn about sex, age, or race from 
the data in Table 6? 
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b. Supervisor opinions 

(1) What conclusions can be drawn from the data in Tables 7-11 
and the supervisor comments? 

(2) What conclusions can be drawn from the data in Table 12? 

c. Are there any overall conclusions, or recommendations, that can 
be made about the project as a result of reviewing the opinions? 



Judgment of the Attainment of Objectives 

1. Program Objectives 

a. Based on evidence of the attainment of program objectives and 
the opinions of project participants and their supervisors, to 
what extent were each of the program objectives attained? 

b. Were there significant obstacles or assets to the attainment of 
program objectives? 

c. What recommendations can be made to improve the attainment of 
each program objective? 

2. Project Objectives 

Answer the following questions for each project . 

a. Based on evidence of the attainment of project objectives and 
the opinions of project participants and their supervisors, to 
what extent were each of the project objectives attained? 

b. Were there any significant obstacles or assets to the attainment 
of project objectives? 

c. What recommendations can be made to improve the attainment 
of each project objective? 
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RESPONSIBILITIES OF THE STATE COORDINATOR 
OF PERSONNEL DEVELOIMENT AT THE 
STATE EVALUATION TEAM MEETING 
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The state coordinator of personnel development has three main 
responsibilities during the meeting of the state evaluation team (SET). 

He should: (l) distribute materials to SET members, (2) begin the 

meeting with an introduction, and (3) serve as a resource person for 
SET menibers during the meeting. 

The state coordinator should distribute to each team member copies 
of the necessary materials fo* the meeting. The materials include those 
described in the main body of S®":2 and any additional materials or data 
supplied by the state coordinator. Also, the state coordinator should 
have ready, in case it is requested by the SET, the Guidelines for 
Developing Behavioral Objectives . The state coordinator should f inally 
provide supplies, clerical assistance, and extra copies of the materials 
as needed. 

The state coordinator should open the meeting with an introduction, 
to include the following: 

(1) An explanation of the purpose of the meeting. The Overview and 
the Manual of Instructions for the State Evaluation Te nm (M-3 ) 
may be used as aids. 

(2) Background information on personnel development activities 
within the state. 

(3) A brief description of the materials distributed to SET members. 
In this description, the state coordinator should specify that 
SET members read M-3 first and that SET members will use M-3 
and the program and project data folders most during the meeting. 
Further, the state coordinator should specify that project 
abstracts sent to SET members before the meeting refer to the 
project data folders provided during the meeting. 

The state coordinator should then leave the meeting, and come back 
only as requested. He should serve as a resource person for the SET and 
supply whatever materials or data the team requests. The state coordinator 
does not have a voice in decisions of the team. 
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Vocational Education Personnel Development 
STATE PROGRAM COORDINATOR SURVEY 

I. DETERMINATION OF PERSONNEL DEVELOPMENT NEEDS 

Please complete the following table by ranking in order of importance 
these five means of determining personnel development needs (l being 
most important, 5 least important). 



Order of 
Importance 


Means of Determining Needs 




A. Consultation with persons in agencies (e.g. , local education 
agency, university). List title and/or function of people 
consulted and their agency: 




B. Review of personnel development literature. List main sources: 




C. Needs generated by previous and/or ongoing projects within your 
state. List projects: 




D. Your own observation and/or experience. Explain: 




E. Other (Specify) 
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II. NEEDS AND PRIORITIES 

In section A of the chart below, please list the stated personnel 
development needs in order of priority. In section B , for each need 
listed in A, rank in order of importance the criteria used in 
establishing the priority of that need (l being most important). 

Rank only those criteria that were used. See the example in 
section A and B of the chart. 

If additional space is required, continue on page 3* 



SECTION A 


SECTION B 


Personnel Development Needs 
Listed in Order of Priority 


< 


Criteria for Setting Priority 


Resources 

(financial, 

personnel) 


Opinions of 
state voc. ed. 
personnel 


Public policy 
(federal, state) 


; Existing or pro- 
jected shortage 
of personnel 


Advice of state 
advisory council 


Manage ab ility 


Other sources 
(specify) 


(EXAMPLE) 

Inservice Training for 50 Local School 
Voc. & Tech. Education Curriculum 
Specialists 




5 


(EX 

1 


ample) 

4 




2 


3 

Local 

school 

supt. 


* 

















(continued) 
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SECTION A 



SECTION B 



Criteria for Setting Priority 



Personnel Development Needs 
Listed in Order of. Priority 
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C. Because of various constraints (for example, lack of available 
resources, state regulations, etc.), certain needs maybe reduced 
in priority. State the reduced priority needs and those constraints 
that, keep the priority low. 
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Other sources 
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III. PROCEDURES FOR SELECTING PROJECTS 

A. How many and what percentage of the proposals for personnel 
development projects designed to meet the program objectives 
were from the following organizations or agencies: 



Agencies 


Total 

Proposals Submitted 


Total 

Proposals Funded 


Universities 


No. 





No. 













Local education agencies 










Community or junior 










Private R & D agencies 








- 


State education agencies 









































B. List the steps used to review proposals for personnel development 
projects. 



Steps in the Review 


People Involved 
(Title and/or Function) 


r ■ 





i 

i 
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C. Section D of the chart below lists criteria that may have 
been used over the total process of screening proposals. 
Check the criteria that were used (Section E) and rank those 
criteria in order of importance (Section F) 0 . being most 
important) . 

If criteria vary by program objectives, reproduce a chart for 
each objective and attach the charts to this instrument. 



SECTION D 


SECTION E 


SECTION F 


Screening Criteria 


Criteria 

Used 


Rank 


Relationship to state program objectives 






Estimated cost of proposal project 






Number of students (participants) involved 






Students (from participant's school or 
agency) for whom the project is aimed 






Past record of institution or agency 
(success with previous projects) 






Proposed project duration 






Feasibility of projects 






Ability and availability of personnel 






Other criteria used (Specify) 

























IV. OBJECTIVES OF PERSONNEL DEVELOPMENT PROGRAM 

Personnel development programs are designed to meet a number of 
specified objectives. Please list the objectives of the personnel 
development program in your state. For each objective, present 
the evidence of accomplishment for that objective. If the evidence 
is in a document, please provide a reference for the document and 
attach a copy of it to this instrument. Use page 6 and attach 
additional sheets if necessary. 



0 
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(Objectives and Evidence of Attainment) 



V. ADDITIONAL COMMENTS 



2 



4. For each of the instructional activities in the table, please 
check the appropriate space to indicate the extent to which 
the activity was used, or if it was not used. Use the space 
below the table to explain simulated or actual work experiences. 



Used 


Not 

Used 


Instructional Activities 


Much 


Little 








a. Guest lecturers and speakers 








b. Audiovisual aids 








c. Class discussion 








d. Textbooks and reading materials 








e. Actual or simulated work 
experiences related to the 
job (Use the space below 
the table to explain. ) 








f. Others (Please specify) 



5« What vocational and technical education personnel development 
projects would you like to see conducted? 
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6. Please use this page and the next to list each stated objective 
of the personnel development project and give evidence of 
attainment. If the evidence is in a document, please provide 
a reference for the document and attach a copy of it to this 
instrument . Attach additional sheets if necessary. 
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Continue on back page 
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7. Additional comments 
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Vocational Education Personnel Development 



A 

B 

C 



PARTICIPANT SURVEY 



Project 



This survey form applies to the personnel development project listed 
above. Your responses will be used by the state as one indicator of how 
beneficial personnel development projects (such as inservice training) are 
to professionals throughout the state. Do not put your name on this form. 



Please check or complete the following: 



I. BACKGROUND DATA 

1* Age 2. Sex Male 3* Race (Optional) 

Female Black 

Red 

White 

Yellow 

Other (specify) 



4. Briefly describe your present job. 
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S-3 
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II. PERCEPTION OF PROJECT 

Respond to the following questions as they apply to the personnel 
development project stated at the top of the first page of this 
survey form. 

1. State what you perceived to be the purpose of the project. 



2 . 



Has the project been completed? Yes 

If yes, did you complete it? Yes 

If you did not, why not? 



No 



No 



3- For each of the instructional activities in the tables below, 
check only one of the six spaces in the table. 







Worth- 

while 


Not 

Worth- 
while _ 








Instructional 

Activities 
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(1) 


(2) 


( 3 ) 


(4) 


(?) 


(6) 


a. 


Guest lecturers and speakers 














b. 


Audiovisual aids 














c. 


Class discussions 














d. 


Textbooks and reading materials 














e. 


Actual or simulated work exper- 
iences related to the job 














f. 


Others (please specify; 

3 


L 
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4. How worthwhile was the project for you? 
Comment 



WORTHWHILE 

Completely 

Partially 

NOT WORTHWHILE 

Partially 

Completely 



5. How well did the project apply to DID APPLY 

your work? Completely 

Comment Partially 

DID NOT APPLY 
Partially 

* Completely 



6. How well organized was the project? 
Comment 



ORGANIZED 

Completely 

Partially 

NOT ORGANIZED 

Partially 

Completely 



7 . Would you recommend this project 

to someone else? Yes 

If no, why not? No 



Please complete the back of this page 






8. What was most relevant about the project? 



9. What was least relevant about the project? 



10. What vocational and technical education personnel 

development projects would you like to see conducted? 
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11. Additional Comments: 
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SUPERVISOR SURVEY 



This survey form is designed to obtain information about the personnel 
development project(s) in which one or more of your employees recently par- 
ticipated. This page is designed to gather information about you, the 
participant’s immediate supervisor, and your agency. The back page requests 
information about the participant (your employee) and the project in which he 
participated. Please complete this page for yourself and the back page 
for the employee listed at the top of that page. Then complete the additional 
attached pages (copies of the back page), if there are any, for each of the 
employees and projects listed at the top of those pages. Your responses will 
be used by the state as one indicator of the effects of personnel development 
projects (such as inservice training) upon professionals throughout the state. 

Please check or complete the following: 

I. DESCRIPTIVE INFORMATION 

1. What is your title? 

2. With which of the following agencies are you associated? 

College or University 

Community College or Technical Institute . 

______ Intermediate School Unit (Regional, Area, or County Unit) 

Local School District 

Stbt? Department of Education 

Others (Specify) 



3. If you are associated with a local school district what is 
its size? 

Less than 10,000 pupils 

10,000 - 20,000 pupils 

20,001 - 50,000 pupils 

More than 50,000 pupils 

4. What vocational and technical education personnel development 
projects would you like to see conducted? 
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Employee’s (Participant's) Name 




Project 

II. PERCEPTION OF PROJECT 



1. How well did the project apply to the participant's did APPLY 

wor ^- Completely 

Partially 

DID NOT APPLY 

Partially 

Completely 

2. Has there been a noticeable improvement in the job 
performance of the participant after he participated 
in the project? 

If SO; how has he improved? _____ Much improvement 

Some improvement 

No improvement 

3. Would you encourage other employees of your agency 
to participate in a similar project? 

Please explain. Yes 

No 



4. Has the participant shared his experiences with 
you, his immediate supervisor? 



5. Has the participant shared his experiences with 
other employees with whom he works? 



6. What provisions were made for the participant to 
share his experiences? 



Yes 

No 



Yes 

No 

Do not know 



Group seminars 
Newsletters 
Informal discussions 
Others (specify) 



III. ADDITIONAL COMMENTS 
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o 

ERIC 



